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IBSIHACT ^ 

1 conpetency-baaad asseasnent procedure for the 
training and selection o* preschool teachers has beeE developed using 
the content validation strategy anfl a variety of Beasurenent 
technique Tlose certified under this procedure nill be called Child 
Developaent Associates. A list of competeiicies, geaeratea and agreea 
upoE by a large nunber of Child Development experts, is the starting 
point for the assessment procedure- The competency list i*as developed 
uad«p the Departaeat of Health, Bducation, and Helfare, Office of 
Chiia Developnent^ by a task force, the Child Developneiit consoitiun, 
incorpoEated. Tlkm task force establisheii siic aajoE areas of 
responsibility for preschool teachers, developed a preliminary set of 
competencies under each of ' six *4najor areas and added a set of 
personal capacities necessary for relating to children. The 
competencies vere then* validattd by means of a stirvey of 
oractltioners and others in the field. An interdiBci|)linary tean 
kpp roach was used which night be a nodel for the aevelopmen-t of other 
c&ipetency- based assessment procedurefi. Hethodolqgy is described in 
teris of pro^^ and step-by-step proceaures 

used. A fe* examples of competencies are appenaixed. CAnthor/HS) 
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COMPETENCY ASSESSMENT PROCEDURES 
FOR TRAINIWG AND CERTIFYrNG THE CHILD DEVELOPMENT ASSOCIATE 

Joy M, Wenne, Shirley Karas^ Dontia C* Nelson, and J, W. Menne 

^ ^Abs tract 

A coTtipetaTicy'^based assessnieTit procedure for the training and selec-- 
tion of pre-school teachers has been cjeveloped using the content vali- 



dation strategy and a variety of measureinent techniques, An Inter^ 
disciplinary teani approach was used in a manner that could be raasonably 
and successfully employed in the development of other competency^baBed 
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assessment procedures^ 



COMPETENCY ASSESSMENT PROCEDURES 
FOR TRATNINf; AND CERTIFYINC; THE CHILD DEVELOPMENT ASSOCIATE 



Joy M. Menne, Shirley Kjiras, Donna C. Nelson, and John W. Honne 



PROBLIiM 



The problem herein is to develop a CQmpel:enc./-based certification pro- 
cedure for pre-school taachers, such as those teachers in Head Start. 
Those certified will be called Child Development Associates (CDAs) . The 
move toi.-ard CDA certification is an attempt Co Increase the quantity and 
Improve the competency of pre-school teachers. Assessment procediires are 
to serve both the diagnostic purpose of indicating the type and extent ot 
training needed by an individual CDA candldaCe and the certlf Icatton pur- 
nose by indicating that the pre-school teacher is, in fact, competent 1^ 
this "new" profession. ThouBh the CUA trainee may acquire coUege credits 
and possibly even a degree, the credits are not to be requirements and, in 
fa/t, may be irrelevant to the certification process. 

A list of competencies, generated and agre*ed upon by a large number of 
Child DevelDpnient "experts", is Che starting poln% for the assessnient pro- 
cedure. This compecency list was developed under the iinpetua of the v.. 
Department of Health, Education and Welfare, Office of Child Development by 
a ta-k force called the Child Development Consotftium,. Incorporated, Ihe 
task force established six major areas of responsibility for pre-school 
teachers, developed a preliminary sat of competencies under each of the six 
major areas and added a set of personal capacities necessary for relating 
to children. The task force then had their set of competencies validated 
by means of a survey of practitloneri and appropriate others in the field, 
obtaining ratings and suggestions. (See Appendix A for examples of the 
competencies,) 

Various early childhood programs throughout the country have been 
charged with the responsibility for developing craining programs and assess 

- ment devices for the CDA, based on th« resulting set of competencies. The 
development of assessment procedures r^'porCed herein Is being sponsored by 
the Iowa Head Start Training Program headquartered in the DeparCinent of 
Child Development at Iowa State University. Although the assessment fro- 

■ cedures are being locally developed, they are expected ulttinattly to meet 
rainiinal, though yet undefined, national cflteria. \^ 



METHODOLOGY 
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'ITiere are tv/o niajor aspects to the methodology being tatllt^ed in meet- 
ing this problem. One iniportant aspect 1.^ tha interdisciplinary profesalon 
aJ uKpt^rttHti of Lhci pur.^oiia attacking thin problem and the roloB of Qach in 
t iIh imdertaklng. The second aspect is the st:ep-by--r>tep procedures follow-- 
6d by each prafessional and the team as a whole. 



ProfGBfjional Exp e r 1 1 b e and R o 1 e s 

There were four specific professional competencifis Involved as 
fallows I 

1) A professional In the area of Educational and Psychological Meas- 
ureniant, who outlined procedures, instructed others, TnoTjltored processes, 
and provided technical support, h. 

2) A Counseling psychologist with item--\^ltlng skills, who functioned 
mainly as a coordinator and group facilitator, 

3) An admlnlBtrator responsible for a state-^'wlde staff training pro^ 
gram and liason with the University Department of Child Developments 

4) A university Instructor and pre^schQOl teacher /trainer , who work- 
ed with Child Development faculty , graduata students and pre-school 
Cfeachers . 

The latter group met almost weekly for many itionths with the counseling 
psycholdgiat for purposes of reviet*^, dlscusiton and asslgnnients , The maas- 
urement person and the administrator met with the group as needed. This 
procedure has insured the appropriateness and relevance to the assessment 
procedure while providing "in.'-serviGe. measureinent training" to the content 
area specialists. 



Step-'by-^step Procedures 

1) For^l ratings of the original list of coinpetencies on an 
"Importance-' scale were made by local Child Dev^ilopinent faculty and graduate 
students and by Head Start trainera,. t^y? Bicause this gave us a basis for 
Judging which of the competencies we should place most emphasis on or those 
vhlch should be eliminated, and it gave a basis for weighting different 
parts of the final product* , 

2) The group discussed each competency In detail for purposes of 
clarification and assessment approach. That ISj the competency statements 
^mtm carefully reviewed in order to Interpret and clarify their meaning, to 

" ■ - 4 
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assess overlap In statements, to consider what types of hehavlors or know- 
ledge would demonscrate each of thrS conipatenciesi and^ subsequently, to 
consider the way(s) in which a glvein competency best be aj^sessed* In other 
words, could a particular uompetenay best be assaased through muJtiple- 
choice Items p true-'false statements, written short-answer itesiui, structured 
axperlenceB (to be video«taped and rated) or some combination thereof. 

3) He5ct| the content area specialists were asked to begin indepen-' 
dently to write out asoesBment ItemB and procedures as well as forrnal ans- 
wexs for each with formal moring or rating procedures or guidelines to be 
used by an evaluation team* liiey were asked also to define the minimal 
acceptable level of each competency. At the beginning of thia process^ 
Individuals were not assigned specific competencies to work on. We believ-- 
ed that more enthusiasm and confidence would be generated for the task if 
each person began by warklng on those competenclss with which he/she felt 
moat capable or Interested in. This also helped to build up an item pool 
so that we would later have more to select from. 

The group met with the counseling piychologtst about every two to four 
weeks to critically review the Items (or procedures) generated, the answers 
and the guidelines. Some items were thereupon ellminatedj others revleedi 
After several meetings^ we began to assess which specific competencies ware 
lacking items and individuals were then assigned to work on those competen- 
cies * 

Efforts to date have been focused primarily upon the written assess- 
ment, At .the time of this writing, only the basic elements of the struct-- 
ured experiences have been set forth and work on the details is just be- 
ginning. Some eKamples of the structured eKperiences to b*2 used are as 
follows : 

a) The trainee will be charged with responsibility for planning and 
supervising classroom activities for perhaps two separate half days* The 
trainee will be given some dlrictlons as to the nature of uhe actiyltiee 
he/she sho'mld plan for. This will be done in order to standardize activi- 
ties across trainees and the competencies which are likely to be demonstra- 
tqd "on the Job"* These classroom sessions will be videQ-^taped; judges will 
later view the tapes and, following the guidelines set forth, will rate the 
trainee's performance on the appropriate competencies, 

b) A number of carefully selected actual or role-^played situations 
will be video-taped . These may include such things as children on a play- 
ground engaged in disruptive or dangerous activities or a film of an irrate 
parent, being critical of the progrMa. The trainee will be shown these 
brief films and asked to respond with how he/she would handle these 
situations p ■ ' 

4) Written items were carefully selected and compiled into a two- 
part assessment tool. Part I was mde up of the objective multiple-choice 
and true-false items: Part II was made up of the written short-answer 
items which will be judged by an evaluation team. 



5 



4 



5) A trial administration of both parts of the aaeeasinerit procedut-i 
wa& carried out with a small group of graduate students, faculty members 
and Head Start trainers , who had not been previously iKtvolved In the pro- 
ject. These testGes were asked to answer the ltems> to critically review 
the items for clarity of nteanlng and appropriateness, to offer suggestions 
and to note the atnount of time it had taken them to answer the iter.a. Feed- 
back from the testfces was utilised' to re-edit the items. After re-editing^ 
thf:^ asHGseraent procedure was again administered to a different and larger 
group of testeesi it waa then re-edlted again, 

6) The objective items were administered to 79 Head Start personnel 
and to 42 flrst«year college studante in a Child Development course, The 
32 items had a ficore range from 14 to 31, a mean score (for all 144 sub- 
jects) of 24.9 and a standard deviation of 3*6, The TO-ZO reliability e'it- 
iraate was 0*68. The iteni-score correlation data indicated that onl3r one 
item needed cjarif ication t)r re-wrlting* The' average score for the Bead 
Start personnel (24,7) was slightly above the average score foT the first- 
year college students .(23.9). '^^Che-scores of the Head Start personnel vJere 
considerably more variable than the scores of the students (standard 'devia- 
tion of %,0 versus 2.8), Some of the Head Start personnel were directors 
and trfliners, which accounted for 24 Hiad Start personnel scores of 28 and 
above, whereas, only six Gtudenta received a score .of 28 and no studetit 
received a score above 28, On the other hand, six Head Start personnel 
received scores below 18i no student scores below 18, 

This data is interpreted to indicate that . the objeative Iterds are 
probably measuring in a suitable way, and the only way to improve thi re-- 
liability of this part of the assessment prDcedure subBtantlally w'ould be ' 
to Increase' the number of items. However, as the objective Items are only 
a part of the assessment procedure * it was decided that a 101-20 reltabillty 
of about O.VO for this part of the assessment would be satisfactory. 

7) The written short-answer Items are being Independently Judged by a 
four-person tei*m using the model answers or guidelines previously written 
as criteria, At this writing, ' this task Is not complete, The reliability 
of the scorin g for this aspect of the assessment la being eatlmiated by 
intercorrelatlng the ratings by the judges for all items and f^r all sub- 
jects, then using the Spearman-Brown average Inter^item procedure to^sti- 
mate the reliability of the average rating, where the average Is the sum of 
the points assigned to a particular subject on a parclcular Item dlv^lded by 
the nimber of raters or judges. This estimated reliability uses the form- 

Y^l^-^^^^^y where "n" Is the numher of raters or judges and '-r" is 

the average of f-ilagnol correlation between the raters, 

8) The final step that will be used to definitely establlih the con- 
tent validity of the entire assessment procedure is to use si^ to 12 
'ekperts" who, as a first step, will be asked to document in wrltimg their 

qualifications (education and eKperience) as experts. This pmeL will then 
be furnisbed wltfh the entire list of objectives to be assessi-d and a copy 
of the complete assessment procedure. They will be asked to Judge Itidtpen- ' 
dently in two waysi , = 
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a) They will first indicate for each item or element of the assess- 
ment procedure the objectlve(s) they believe is being measured, 

b) They will then rate each item ox eleinent on a five-point ecale 
from "completeJ.y inappropriate" to "niost appropriate" for measuring the 
objective which they indicated in "a*^ as being measured, 

Hopofullyi there will be a strong consensus about the objective bolng rneas^ 
ured by each item or element of the aaiesgment procedure and the flve-^point 
"approprtateneSB ' rating for each IteTn Qt eleinent will average 4*0 or abovei 
One of the authors has utilised these same stepwise procedures for develop- 
ing an assessment procedure and formally establishing content validity for 
Dver 250 classes of jobs. In every casep the above criteria have baen inet. 



UTILIZATION 



Throughout'* the develbpmeht of the assessment procedure ^ a minimum 
acceptable competancy has been defined for each objective, Theteforaj in 
using the asdessmant procedure in the certification process ^ it will ba re- 
quired that the candidate score above the minimal acceptable level on every 
coinpetency.. assessed. It is recognized that the reliability of the aisess- 
ment procedurea for measuring individual competencies will be low when there 
ire very few assessment Itans or prqcedtires that pertain to one eottipetency. 
However, as it would be a much more expensive and tlme-consumlni task to 
develop procedurea that would have high reliability for each objective raeas- 
ured (both in development and in adminiitratlon) , it seems more fiasibla to 
require that the candidate score abovei the minimal level on every compete-- 
ncy and that the combined score over all competencies be set for passing at 
a level at least one standard error of measurement above ^the score that 
would be obtained if the candidate attained only the minimal level on every 
competency. 

The assessment procedure would also be used to determine needed areas 
of training* In this way, the procedure is administered ^ as a "pre^test" 
as well as a final test, with the training effort being focused on ^,he areas 
of greatest need as indicated by the Initial assessment. 

The assessment procedure is not designed to be administered in one 
session. Rather, both the initial assessment of the new candidate prior to 
training and the final assessment for certification can be admlnlitered over 
a period of several days, a week, or even more. Then top, on completion of 
a certain area of tralrilng, that part of the assessment procedui^e may again 
be administered before the candidate moves on to focus on; another training 
area. 
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DISCUSSION 



It is feaslblt* Co develop competency-baaed as^isssment procedut^^ which 
caii meet profeesional Btandards for reliability and validity* The resulting 
assessment procedure has many of the desirable attributes espoused thorn 
opposing "criterion-referenced" testing or asaessimant. However, tha v^li'd* 
ity of the procedure is determined by the age-old ^'oontent*' methodp utiliis^ 
Ing a formal and independent judgtnent proceas by experts, Alsoj tha re« 
llnbiltty of the objictlve aspects of the aipessmeat Is being estimated by 
procedures commonly used in norm-referenced testa. Whan the assessmnt 
product Is used in the certification process^ however, it is not eKp^atad 
that the reliability 3, if again estiJnatedp will be m hlgh| it is eKp^atad 
that the variaijca of the scores obtained by the ceictlf Icande will quite 

Although the procedure for developing the asaeisment tool is quitM 
straight-forward, the process^ is not easy | 'it Is time-consimlng and ^:?Qpan-^ 
sive. In order to develop a quality assessment product , it Is requiirad 
that content area spacialiats become more knowledgeable in the area 0f 
measurement and the idieasurement specialists be seiiiitlve to the need^ mi 
eKpertlse of the content area specialists. The actual. group process Is 
most important. The participants need to perceive a sense of learning acid 
accomplishment* thus progress, The team, felt that preclpltoualy setting 
target dates could result in a lower quality product, yet It was felt tti^t 
It was necessary to Ueep moving so as to perceive progress. 

In a way,, the greatest frustration with this effort has been th^ lack 
of understanding on the part of federal and regional officials. Asking 
each locality to develop its own assessment procedure may smm deslrabla tp 
many people; however, every locality Is not likely to have the profe$atonaX 
reeources necessary In order ^ to develop a quality product. Further, thl$ 
work has been undertaken with the understanding that the assessment pro^ 
Gedures must meet as yet undefined minimal criteria to be specified at the 
national, level. In fact, it is not known (and even ieems sbm^hat: unlikely) 
that these minimal criteria will concern the psychometric qualities of tha 
aaseasnient procedure and/or developtnental methodology. 

The group seems well satisfied with the effarte and pTfogress and is 
anxious to complete the project. They feel that tha end is in sight and 
that the result will be a quality product. 
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APPENDIX A 
^EXAMPLIS OP CQMPETENCIES 



In the Bxamples given beloWj those items underscored are some of the 
major areas of responsibility listed for pre-school teacheri. Each area * 
presented herein is followed by m example of a, competency Item or state- 
ment within that area of ^respoTisibillty * ^ 

'"Advancl n | Physical and I ntellectual Competence" 

■'Provide varied oppOTftumities for children's active participations 
independent cholceap experimentation and problem^solvlng within the context 
of a structured, organized setting and programi'' 

j 

■'StlTTiulate focused aetivitlesi observing^ attending , inltiatingi 
carrying through^ raising questloni^ searching answers' and solutions for the 
real problems, that are encountwad and reviewing the outcomes of experience.-' 

"Building Positive 5elf-cpricg |>t and Individual Strength" 

■ ^ 

"Give direct, reallstta affirmation to the child's advancing 
skills, growing initiative and resporialbillty, increasing capacity for 
adaptation^ and einerging interest in. cooperation^ in terms of ,the child's 
acttial behavior. ■' > , 



EKamplen of personal capacities essential for pre-school teachers: 

*'To hm sensitive to childrw'a feelings and the qualities of young 
thinking"^ ^ ; ' ^ . 

"To ba able to protect ord^yllness without aacrlficing spontaneity 
and child-like exuberance" 

"To be able to exercise comtrol without being threatening" 



